Smart necklace for social awareness

ABSTRACT

A device for providing feedback includes an imaging device configured to capture an image of a subject, a feedback device, and a controller communicatively coupled to the imaging device and the feedback device. The controller includes at least one processor and at least one memory storing computer readable and executable instructions that, when executed by the processor, cause the controller to determine one or more identifying characteristics of the subject, set one or more parameters for determining a facial expression of the subject in the image based on the one or more identifying characteristics of the subject, determine the facial expression of the subject in the image based on the one or more parameters, and provide the feedback with the feedback device based on the determined facial expression.

TECHNICAL FIELD

The present disclosure generally relates to devices for providingfeedback and, more specifically, to devices that provide feedback basedon one or more identifying characteristics of a subject proximate to thedevice and/or a level of a facial expression of the subject.

BACKGROUND

A person with impaired vision may not be able to recognize a facialexpression of a person near to him or her. Similarly, a person on theautism spectrum may not be able to properly interpret a facialexpression of a person near to him or her. Facial expressions can varydepending on people. For example, a smiling facial expression of personA is different from a smiling facial expression of person B. It may bedesirable to provide facial expression recognition that depends on asubject being analyzed. In addition, it may be desirable to provideinformation on the level of a facial expression made by a person.

Accordingly, a need exists for devices that provide feedback based onthe identifying characteristic of a person and/or the level of a facialexpression of a person.

SUMMARY

In one embodiment, a device for providing feedback includes an imagingdevice configured to capture an image of a subject, a feedback device,and a controller communicatively coupled to the imaging device and thefeedback device. The controller includes at least one processor and atleast one memory storing computer readable and executable instructionsthat, when executed by the processor, cause the controller to determineone or more identifying characteristics of the subject, set one or moreparameters for determining a facial expression of the subject in theimage based on the one or more identifying characteristics of thesubject, determine the facial expression of the subject in the imagebased on the one or more parameters, and provide the feedback with thefeedback device based on the determined facial expression.

In another embodiment, a device for providing a feedback includes animaging device configured to capture an image of a face of a subject, afeedback device, and a controller communicatively coupled to the imagingdevice and the feedback device. The controller includes at least oneprocessor and at least one memory storing computer readable andexecutable instructions that, when executed by the processor, cause thecontroller to process the image of the face to determine one or morefacial expression parameters, determine an emotion of the face based onthe one or more facial expression parameters, classify the emotion as apositive emotion or a negative emotion, send to the feedback device aninstruction for vibrating a first side of the device with the feedbackdevice in response to the emotion being classified as the positiveemotion, and send to the feedback device an instruction for vibrating asecond side of the device with the feedback device in response to theemotion being classified as the negative emotion.

In another embodiment, a method for providing a feedback includescapturing an image of a subject, determining one or more identifyingcharacteristics of the subject, setting one or more parameters fordetermining a facial expression of the subject based on the one or moreidentifying characteristics of the subject, determining the facialexpression of the subject in the image based on the one or moreparameters, and providing, by the processor, the feedback with afeedback device based on the determined facial expression.

These and additional features provided by the embodiments of the presentdisclosure will be more fully understood in view of the followingdetailed description, in conjunction with the drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

The embodiments set forth in the drawings are illustrative and exemplaryin nature and not intended to limit the disclosure. The followingdetailed description of the illustrative embodiments can be understoodwhen read in conjunction with the following drawings, where likestructure is indicated with like reference numerals and in which:

FIG. 1A schematically depicts an exemplary embodiment of variouselectronic components of a device for providing feedback to a user inaccordance with one or more embodiments shown and described herein;

FIG. 1B depicts a front view of a device in accordance with one or moreembodiments shown and described herein;

FIG. 2 depicts a flow chart of a method for providing feedback using adevice in accordance with one or more embodiments shown and describedherein;

FIG. 3 depicts a flow chart of a method for providing feedback using adevice in accordance with another embodiment shown and described herein;

FIG. 4A schematically depicts an array of a feedback device inaccordance with one or more embodiments described and shown herein;

FIG. 4B schematically depicts a vibrating pattern of an array of thefeedback device in accordance with one or more embodiments described andshown herein;

FIG. 4C schematically depicts a vibrating pattern of an array of thefeedback device in accordance with one or more embodiments described andshown herein;

FIG. 4D schematically depicts a vibrating pattern of an array of thefeedback device in accordance with one or more embodiments described andshown herein;

FIG. 4E schematically depicts a vibrating pattern of an array of thefeedback device in accordance with one or more embodiments described andshown herein;

FIG. 4F schematically depicts a vibrating pattern of an array of thefeedback device in accordance with one or more embodiments described andshown herein;

FIG. 4G schematically depicts a vibrating pattern of an array of thefeedback device in accordance with one or more embodiments described andshown herein;

FIG. 5A depicts a graph illustrating a relationship between a level of afacial expression and a vibration intensity of the feedback device inaccordance with one or more embodiments shown and described herein; and

FIG. 5B depicts a graph illustrating a relationship between a level of afacial expression and a vibration intensity of the feedback device inaccordance with one or more embodiments shown and described herein.

DETAILED DESCRIPTION

The embodiments disclosed herein include devices that provide feedbackthat is determined based on one or more identifying characteristics of asubject, and vary feedback based on the type and/or level of a facialexpression of a person. Referring generally to FIG. 1A, a device forproviding feedback includes an imaging device configured to capture animage of a subject, a feedback device, and a controller communicativelycoupled to the imaging device and the feedback device. The controllerincludes at least one processor and at least one memory storing computerreadable and executable instructions that, when executed by theprocessor, cause the controller to determine one or more identifyingcharacteristics of the subject, set one or more parameters fordetermining a facial expression of the subject in the image based on theone or more identifying characteristics of the subject, determine thefacial expression of the subject in the image based on the one or moreparameters, and provide the feedback with the feedback device based onthe determined facial expression.

Some assisting devices do not provide varying haptic feedback dependingon the identity of a person proximate to the devices. In addition, theconventional vision assist devices do not provide information on thelevel of a facial expression made by a person. For example, a user ofthe conventional assisting device cannot recognize a level of smiling ofa person in front of the user. The embodiments described herein overcomethese limitations by providing varying feedback based on a level of afacial expression of a person.

Referring now to the drawings, FIG. 1A schematically depicts anexemplary embodiment of a device 100 for providing feedback to a user inaccordance with one or more embodiments shown and described herein. Thefeedback may be a tactile feedback, an audible feedback, a visualfeedback, etc. The device 100 may be a smart device including, notlimited to, a smart necklace device, a smart glasses, a smart helmet, asmart earing, etc. The device 100 includes a controller 102, a feedbackdevice 130, an imaging device 140, network interface hardware 150, acommunication path 160, an audible feedback device 170, and a microphone172. The various components of the device 100 will now be described.

The controller 102 includes a processor 110 and a non-transitoryelectronic memory 120 to which various components are communicativelycoupled, as will be described in further detail below. In someembodiments, the processor 110 and the non-transitory electronic memory120 and/or the other components are included within a single device. Inother embodiments, the processor 110 and the non-transitory electronicmemory 120 and/or the other components may be distributed among multipledevices that are communicatively coupled.

The device 100 includes the non-transitory electronic memory 120 thatstores a set of machine readable instructions. The processor 110executes the machine readable instructions stored in the non-transitoryelectronic memory 120. The non-transitory electronic memory 120 maycomprise RAM, ROM, flash memories, hard drives, or any device capable ofstoring machine readable instructions such that the machine readableinstructions can be accessed by the processor 110. The machine readableinstructions comprise logic or algorithm(s) written in any programminglanguage of any generation (e.g., 1GL, 2GL, 3GL, 4GL, or 5GL) such as,for example, machine language that may be directly executed by theprocessor 110, or assembly language, object-oriented programming (OOP),scripting languages, microcode, etc., that may be compiled or assembledinto machine readable instructions and stored in the non-transitoryelectronic memory 120. Alternatively, the machine readable instructionsmay be written in a hardware description language (HDL), such as logicimplemented via either a field-programmable gate array (FPGA)configuration or an application-specific integrated circuit (ASIC), ortheir equivalents. Accordingly, the methods described herein may beimplemented in any conventional computer programming language, aspre-programmed hardware elements, or as a combination of hardware andsoftware components. The non-transitory electronic memory 120 may beimplemented as one memory module or a plurality of memory modules.

In some embodiments, the non-transitory electronic memory 120 includesinstructions for executing the functions of the device 100. Theinstructions may include instructions for determining a type of a facialexpression, instructions for determining a level of a facial expression,instructions for determining a pattern of feedback, and instructions fordetermining an intensity of feedback. The instructions for determiningtype of a facial expression, when executed by the processor 110, maydetermine a type of a facial expression of a captured image. The typesof facial expressions may include happy, smile, sad, surprise, angry,fear, disgust, etc. The instructions for determining a type of a facialexpression may include an image recognition algorithm.

The instructions for determining the level of a facial expression, whenexecuted by the processor 110, may determine a level of a facialexpression of a captured image. For example, once the type of a facialexpression is determined as a smiling facial expression, theinstructions for determining a level of a facial expression maydetermine the level of smiling of the facial expression. The level mayindicate between 0% and 100%. The level of smiling will be described infurther detail below with reference to FIG. 2.

The instructions for determining a pattern of feedback, when executed bythe processor 110, may determine a pattern of feedback based on a typeof a facial expression. For example, different patterns of feedback maybe used for different facial expressions, such as smiling and angryfacial expressions. The instructions for determining an intensity offeedback, when executed by the processor 110, may determine an intensityof feedback based on the level of a facial expression. For example, theintensity of vibration feedback of the device 100 may be proportional tothe level of a facial expression. The functionality of each of theseinstructions will be described in further detail below.

The processor 110 may be any device capable of executing machinereadable instructions. For example, the processor 110 may be anintegrated circuit, a microchip, a computer, or any other computingdevice. The non-transitory electronic memory 120 and the processor 110are coupled to the communication path 160 that provides signalinterconnectivity between various components and/or modules of thedevice 100. Accordingly, the communication path 160 may communicativelycouple any number of processors with one another, and allow the modulescoupled to the communication path 160 to operate in a distributedcomputing environment. Specifically, each of the modules may operate asa node that may send and/or receive data. As used herein, the term“communicatively coupled” means that coupled components are capable ofexchanging data signals with one another such as, for example,electrical signals via conductive medium, electromagnetic signals viaair, optical signals via optical waveguides, and the like.

Accordingly, the communication path 160 may be formed from any mediumthat is capable of transmitting a signal such as, for example,conductive wires, conductive traces, optical waveguides, or the like.Moreover, the communication path 160 may be formed from a combination ofmediums capable of transmitting signals. In some embodiments, thecommunication path 160 comprises a combination of conductive traces,conductive wires, connectors, and buses that cooperate to permit thetransmission of electrical data signals to components such asprocessors, memories, sensors, input devices, output devices, andcommunication devices. Additionally, it is noted that the term “signal”means a waveform (e.g., electrical, optical, magnetic, mechanical orelectromagnetic), such as DC, AC, sinusoidal-wave, triangular-wave,square-wave, vibration, and the like, capable of traveling through amedium.

As schematically depicted in FIG. 1A, the communication path 160communicatively couples the processor 110 and the non-transitoryelectronic memory 120 with a plurality of other components of the device100. For example, the device 100 depicted in FIG. 1A includes theprocessor 110 and the non-transitory electronic memory 120communicatively coupled with the feedback device 130, the imaging device140, the network interface hardware 150, the audible feedback device170, and the microphone 172.

The feedback device 130 may be any device capable of providing feedbackto a user. The feedback device 130 may include a vibration device (suchas in embodiments in which feedback is delivered through vibration), anair blowing device (such as in embodiments in which feedback isdelivered through a puff of air), or a pressure generating device (suchas in embodiments in which the feedback is delivered through generatedpressure). In some embodiments, the feedback device 130 comprises anarray of feedback devices that provide the user with more detailedfeedback. For example, an array (e.g., a 2×2 array or 3×3 array) offeedback devices can provide different types of feedback to the user.For example, feedback received on a left side of a user may indicate onetype of a facial expression, such as a smiling facial expression, andfeedback received on a right side of a user may indicate another type ofa facial expression, such as an angry facial expression. In someembodiments, the feedback device 130 is wearable on the user, forexample as a necklace, a belt, a wristband, a waist-pack, an adhesive,or a button. In some embodiments, the feedback device 130 is located ina device separate from some or all of the other components of the device100 and communicatively coupled with the device 100.

The imaging device 140 is coupled to the communication path 160 andcommunicatively coupled to the processor 110. The imaging device 140 maybe any device having one or more sensing devices (e.g., pixels) capableof detecting radiation in an ultraviolet wavelength band, a visiblelight wavelength band, or an infrared wavelength band. The imagingdevice 140 may have any resolution. The imaging device 140 may includean omni-directional camera, or a panoramic camera. In some embodiments,one or more optical components, such as a mirror, fish-eye lens, or anyother type of lens may be optically coupled to the imaging device 140.The imaging device 140 may be used to capture an image of a subjectproximate to a user of the device 100.

The device 100 includes network interface hardware 150 forcommunicatively coupling the device 100 to a server 190 (e.g., an imagestorage server). The network interface hardware 150 can becommunicatively coupled to the communication path 160 and can be anydevice capable of transmitting and/or receiving data via a network.Accordingly, the network interface hardware 150 can include acommunication transceiver for sending and/or receiving any wired orwireless communication. For example, the network interface hardware 150may include an antenna, a modem, LAN port, Wi-Fi card, WiMax card,mobile communications hardware, near-field communication hardware,satellite communication hardware and/or any wired or wireless hardwarefor communicating with other networks and/or devices. In one embodiment,the network interface hardware 150 includes hardware configured tooperate in accordance with the Bluetooth wireless communicationprotocol. Some embodiments may not include the network interfacehardware 150.

The audible feedback device 170 may be any device capable of providingaudible feedback to a user. The audible feedback device 170 may includea speaker, headphones, or the like. In some embodiments, the audiblefeedback may be delivered to the user with the speaker or headphones ina 3-dimensional (3D) audio placement format. In some embodiments, theaudible feedback device 170 is integral with the device 100, as depictedin FIG. 1A. In further embodiments, the audible feedback device 170 islocated in a device separate from some or all of the other components ofthe device 100 and communicatively coupled with the device 100. In someembodiments, the audible feedback device 170 is not included in thedevice 100.

The microphone 172 is coupled to the communication path 160 andcommunicatively coupled to the processor 110. The microphone 172 mayreceive acoustic vibrations from a person proximate to the device 100and transform the acoustic vibrations into an electrical signalindicative of the sound. The electrical signal indicative of the soundmay be assessed to determine an identity of a subject and/or one or morespeech parameters, as explained below.

The device 100 may be communicatively coupled to the server 190 by anetwork 180. In one embodiment, the network 180 may include one or morecomputer networks (e.g., a personal area network, a local area network,or a wide area network), cellular networks, satellite networks and/or aglobal positioning system and combinations thereof. Accordingly, thedevice 100 can be communicatively coupled to the network 180 via a widearea network, via a local area network, via a personal area network, viaa cellular network, via a satellite network, etc. Suitable local areanetworks may include wired Ethernet and/or wireless technologies suchas, for example, wireless fidelity (Wi-Fi). Suitable personal areanetworks may include wireless technologies such as, for example, IrDA,Bluetooth, Wireless USB, Z-Wave, ZigBee, and/or other near fieldcommunication protocols. Suitable cellular networks include, but are notlimited to, technologies such as LTE, WiMAX, UMTS, CDMA, and GSM.

While FIG. 1A depicts the processor 110, the non-transitory electronicmemory 120, the feedback device 130, the imaging device 140, and theaudible feedback device 170 and the microphone 172 in a single, integraldevice 100, it should be understood that one or more of these componentsmay be distributed among multiple devices in a variety ofconfigurations.

FIG. 1B depicts a front view of the device in FIG. 1A according to oneor more embodiments shown and described herein. The device 100 may be asmart necklace. In FIG. 1B, the device 100 includes two feedback devices130, two imaging devices 140, the audible feedback device 170 and themicrophone 172. While FIG. 1B depicts two feedback devices 130 and twoimaging devices 140 of the device 100, the device 100 may include morethan two or less than two feedback devices 130 or imaging devices 140.

FIG. 2 depicts a flow chart of a method for providing feedback using adevice in accordance with one or more embodiments shown and describedherein. In step 230, the processor 110 of the device 100 determines oneor more identifying characteristics of the subject. In some embodiment,the processor 110 may determine the one or more identifyingcharacteristics of the subject based on an image of the subject capturedby the imaging device 140. The processor 110 of the device 100 mayimplement image an recognition algorithm on the face in the capturedimage to obtain one or more identifying characteristics of the subject.The one or more identifying characteristics of the subject may includean identity, a gender, an age, a ethnicity or cultural background, etc.In another embodiment, the processor 110 of the device 100 determinesone or more identifying characteristics of the subject based on theelectrical signal output by the microphone 172.

In step 220, the processor 110 of the device 100 sets one or moreparameters for determining a facial expression of the subject based onthe one or more identifying characteristics of the subject determined instep 210. The one or more parameters may include parameters for commonfacial features including, but not limited to, parameters for teeth,parameters for eyes, parameters for an outer lip, parameters for acurvature of the face, etc. The processor 110 may set one or moreparameters for determining a facial expression based on an identity ofthe subject. For example, if the processor 110 identifies the subject asa person A, then the processor 110 may set one or more parameters fordetermining a facial expression with respect to the person A. Thesetting of one or more parameters may include parameters associated withthe person A's smiling facial expression, parameters associated with theperson A's angry facial expression, etc.

In some embodiments, one or more parameters for determining a facialexpression for the subject may be retrieved from the non-transitoryelectronic memory 120. For example, if the processor 110 identifies thesubject as the person A, the processor 110 may retrieve parametersassociated with the person A's various expressions, such as a smilingfacial expression, an angry expression, etc. from the non-transitoryelectronic memory 120. In another embodiment, one or more parameters fordetermining a facial expression for the subject may be retrieved fromthe server 190. For example, if the processor 110 identifies the subjectas the person B, the processor 110 may retrieve parameters associatedwith the person B's various expressions, such as a smiling facialexpression, an angry expression, etc. from the server.

The processor 110 may set one or more parameters for determining afacial expression based on a gender of the subject. For example, if theprocessor 110 determines that the subject is a female, then theprocessor 110 may set one or more parameters for determining a facialexpression of a female. The set one or more parameters may includeparameters associated with average female's smiling facial expression,parameters associated with female's angry facial expression, etc. Theseparameters may be pre-stored in the non-transitory electronic memory 120or the server 190 and the processor 110 may receive the parameters fromthe non-transitory electronic memory 120 or the server 190.

The processor 110 may set one or more parameters for determining afacial expression based on an age of the subject. For example, if theprocessor 110 determines that the subject is a teenager, then theprocessor 110 may set one or more parameters for determining a facialexpression of a teenager. The set one or more parameters may includeparameters associated with average teenager's smiling facial expression,parameters associated with average teenager's angry facial expression,etc. These parameters may be pre-stored in the non-transitory electronicmemory 120 or the server 190 and the processor 110 may receive theparameters from the non-transitory electronic memory 120 or the server190.

The processor 110 may set one or more parameters for determining afacial expression based on an ethnicity or cultural background of thesubject. For example, if the processor 110 determines that the subjectis a person from ethnicity or cultural background A, then the processor110 may set one or more parameters for determining a facial expressionof a person from ethnicity or cultural background A. The set one or moreparameters may include parameters associated with average person fromethnicity or cultural background A's smiling facial expression,parameters associated with average person from ethnicity or culturalbackground A's angry facial expression, etc. These parameters may bepre-stored in the non-transitory electronic memory 120 or the server 190and the processor 110 may receive the parameters from the non-transitoryelectronic memory 120 or the server 190.

In step 230, the processor 110 of the device 100 determines the facialexpression of the subject in the image based on the one or moreparameters set in step 220. The processor may implement a facialexpression recognition algorithm on the captured image to determine afacial expression in the image. The facial expression recognitionalgorithm employs the one or more parameters set in step 220. Forexample, when the subject is determined as a female at the age of 50 instep 220, the processor 110 implements the facial expression recognitionalgorithm using parameters for determining a facial expression of afemale at the age of 50. The parameters for determining a facialexpression of a female at the age of 50 may determine the facialexpression of the captured subject more accurately than parameters fordetermining a facial expression of general people. For example, theprocessor 110 may determine that a facial expression in the image is asmiling facial expression by implementing the facial expressionrecognition algorithm using parameters for determining a facialexpression of a female at the age of 50. In contrast, the processor 110may not accurately determine that a facial expression in the image is asmiling facial expression by implementing the facial expressionrecognition algorithm using parameters for determining a facialexpression of general people because the parameters are not finely tunedto a female at the age of 50.

In some embodiments, when the subject is determined as a person A instep 220, the processor 110 implements the facial expression recognitionalgorithm using parameters for determining a facial expression of personA. The parameters for determining a facial expression of person A maydetermine the facial expression of person A more accurately thanparameters for determining a facial expression of general people. Forexample, the processor 110 may determine that a facial expression in theimage is an angry facial expression by implementing the facialexpression recognition algorithm using parameters for determining afacial expression of person A. In contrast, the processor 110 may notaccurately determine that a facial expression in the image is an angryfacial expression by implementing the facial expression recognitionalgorithm using parameters for determining a facial expression ofgeneral people because the parameters are not finely tuned to person A.

In step 240, the feedback device 130 provides feedback based on thefacial expression. The device 100 may be configured to recognize thetype of a facial expression and the level of a facial expression. In oneembodiment, the device 100 may provide different patterns of vibrationdepending on the type of a facial expression. For example, the device100 may provide a pattern A of vibration when it is determined that thetype of a facial expression in the captured image is a smiling facialexpression, and provide a pattern B of vibration when it is determinedthat the type of a facial expression in the captured image is an angryfacial expression. The pattern A may be different from the pattern B.For example, the vibration pattern A may be a continuous vibration for apredetermined time and the vibration pattern B may be an intermittentvibration. In another embodiment, the device 100 may use differentvibration patterns that correspond to Morse code. For example, thedevice 100 may provide a vibration pattern corresponding to Morse code Awhen it is determined that the type of a facial expression in thecaptured image is smiling, and provide a vibration pattern correspondingto Morse code B when it is determined that the type of a facialexpression in the captured image is an angry facial expression.

FIG. 3 depicts a flow chart of a method for providing feedback using adevice in accordance with another embodiment shown and described herein.In step 310, the imaging device 140 of the device 100 identifies asubject proximate to the device 100. The imaging device 140 may beoperable to sense the location and movement of the subject. The imagingdevice 140 can locate the subject as a whole or can locate more specificsegments of the subject, such as the subject's face.

In step 320, the imaging device 140 captures an image of the face of thesubject. For example, the imaging device 140 of the device 100 may takea photo of a face of a person locating in front of the device 100. Theprocessor 110 of the device 100 may implement an image recognitionalgorithm on the captured image to identify the person. For example, aface recognition algorithm or other conventional image recognitionalgorithm may be used to determine an identity of the subject. Inanother embodiment, the imaging device 140 may capture a gesture of thesubject. For example, the imaging device 140 may capture a hand gesture,a shoulder gesture, a head gesture (e.g., nodding), etc.

In step 330, the processor 110 of the device 100 processes the capturedimage to determine one or more facial expression parameters. The one ormore facial expression parameters may include parameters for commonfacial features including, but not limited to, parameters for teeth,parameters for eyes, parameters for an outer lip, parameters for acurvature of the face, etc. The facial expression parameters may bestored in the non-transitory electronic memory 120 in association withthe identity of the subject. The parameters for teeth may include avalue proportional to the number of teeth shown in the captured image.The parameters for teeth may also include a value related to the size ofteeth exposed in the captured image. The parameters for eyes may includea degree of opening of the eyes, the contour of the eyes, etc. Theparameters for an outer lip may include a degree of opening of the outerlip, the contour of the outer lip, etc.

In another embodiment, the processor 110 of the device 100 may processthe captured image to determine one or more gesture parameters. The oneor more gesture parameters may include parameters for hand gesture,parameters for shoulder gesture, parameters for head gesture. The one ormore gesture parameters may be stored in the non-transitory electronicmemory 120 in association with the identity of the subject.

In step 340, the processor 110 determines an emotion of the face basedon the one or more facial expression parameters. In one embodiment, theprocessor 110 may compare the one or more facial expression parameterswith predetermined facial expression parameters associated with variousemotions, such as happy, smiling, sad, surprise, angry, fear, disgust,etc. For example, the processor 110 may determine that the emotion ofthe captured image is happy, if the one or more facial expressionparameters are closest to predetermined parameters associated with thehappy emotion among predetermined parameters associated with variousemotions. The emotion of the face may be stored in the non-transitoryelectronic memory 120 along with the one or more facial expressionparameters and/or the identity of the subject.

In some embodiments, the processor 110 may retrieve facial expressionparameters associated with the subject that were previously stored inthe non-transitory electronic memory 120. For example, when the imagingdevice 140 captures an image of a person A's face and determines theidentification of the person A, the processor 110 may retrieve facialexpression parameters associated with the person A's happy emotion,facial expression parameters associated with angry emotion, facialexpression parameters associated with the person A's sad emotion, etc.from the non-transitory electronic memory 120. Then, the processor 110may compare facial expression parameters of the captured image withfacial expression parameters retrieved from the non-transitoryelectronic memory 120. If the facial expression parameters of thecaptured image are closest to facial expression parameters associatedwith the person A's sad emotion among the retrieved facial expressionparameters, the processor 110 determines that the type of the facialexpression of the captured image is a sad emotion. Similarly, if thefacial expression parameters of the captured image are closest to facialexpression parameters associated with the person A's angry emotion amongthe retrieved facial expression parameters, the processor 110 determinesthat the type of the facial expression of the captured image is an angryemotion.

In another embodiment, the processor 110 may retrieve facial expressionparameters associated with the subject from the server 190 via thenetwork 180. The server 190 may store facial expression parameters inassociation with an identity of a subject and a type of emotion. Forexample, when the imaging device 140 captures an image of a person B'sface and determines the identification of the person B, the processor110 may retrieve facial expression parameters associated with the personB's happy emotion, facial expression parameters associated with theperson B's angry emotion, facial expression parameters associated withthe person B's sad emotion, etc. from the server 190. Then, theprocessor 110 may compare facial expression parameters of the capturedimage with facial expression parameters retrieved from the server 190.If the facial expression parameters of the captured image are closest tofacial expression parameters associated with the person B's surpriseemotion among the retrieved facial expression parameters, the processor110 determines that the type of the facial expression of the capturedimage is a surprise emotion.

In step 350, the processor 110 classifies the emotion as a positiveemotion or a negative emotion. For example, if the processor 110determines that the emotion of the captured face is happy emotion, theprocessor classifies the emotion as a positive emotion. If the processor110 determines that the emotion of the captured face is sad emotion, theprocessor classifies the emotion as a negative emotion.

In step 360, the feedback device 130 vibrates a first side of the devicewhen the processor 110 determines that the emotion of the captured faceis a positive emotion. The intensity of vibration may be determinedbased on the level of a facial expression. The processor 110 determinesa level of the facial expression of the captured image based on the oneor more facial expression parameters. The processor 110 may compare theone or more facial expression parameters of the captured image withfacial expression parameters associated with various levels of a facialexpression. The facial expression parameters associated with variouslevels of a facial expression, e.g., a smiling facial expression, may bestored in the non-transitory electronic memory 120. In one embodiment,facial expression parameters associated with 0% of smiling up to 100% ofsmiling with an increment of a certain percentage (e.g., 10%) may bestored in the non-transitory electronic memory 120. The facialexpression parameters stored in the non-transitory electronic memory 120may be associated with the identity of a person. If the facialexpression parameters associated with a certain level of smiling areclosest to the one or more facial expression parameters of the capturedimage, the processor 110 may determine that certain level as the levelof the facial expression for the captured image. For example, theprocessor 110 may calculate deviations between the one or more facialexpression parameters of the captured image and facial expressionparameters associated with various levels of smiling. If the deviationbetween the one or more facial expression parameters of the capturedimage and the facial expression parameters associated with 50% ofsmiling is the smallest, the processor 110 may determine that the levelof smiling for the captured image is 50% of smiling.

In another embodiment, facial expression parameters associated with 0%of an angry facial expression up to 100% of an angry facial expressionwith an increment of a certain percentage (e.g., 20%) may be stored inthe server 190. The processor 110 may determine facial expressionparameters associated with a certain level of an angry facial expressionthat are closest to the one or more facial expression parameters of thecaptured image. For example, the processor 110 may calculate deviationsbetween the one or more facial expression parameters of the capturedimage and angry facial expression parameters associated with differentlevels of a facial expression. If the deviation between the one or morefacial expression parameters of the captured image and facial expressionparameters associated with 100% of an angry facial expression is thesmallest, the processor 110 may determine that the level of angry facialexpression for the captured image is 100% of an angry facial expression.

The vibration intensity of the feedback device 130 may be proportionalto the level of the facial expression. For example, the processor 110may determine the vibration intensity of the feedback device 130 as 50%of the maximum vibration intensity if the determined level of smiling is50%, and determine the vibration intensity of the feedback device 130 as100% of the maximum vibration intensity if the determined level ofsmiling is 100%. In another example, the processor 110 may determine thevibration intensity of the feedback device 130 as 30% of the maximumvibration intensity if the determined level of angry facial expressionis 30%.

In some embodiments, the processor 110 may determine a vibrationintensity of the feedback device 130 further based on speech parametersincluding a volume of speech from the microphone 172. For example, theprocessor 110 may determine a vibration intensity of the feedback device130 based on a weighted average of the level of the facial expressionand the level of volume of speech. If the level of smiling is 70% andthe level of volume of speech is 30%, the vibration intensity of thefeedback device 130 may be calculated as α×0.7+(1−α)×0.3. The parametera may be a predetermined value.

In step 370, the feedback device 130 vibrates a second side of thedevice when the processor 110 determines that the emotion of thecaptured face is a negative emotion. Similar to step 360, the intensityof vibration may be determined based on the level of a facialexpression.

FIG. 4A schematically depicts an array of the feedback device 130 of thedevice 100 in accordance with one or more embodiments described andshown herein. The feedback device 130 may include an array of vibratingdevices. For example, the feedback device 130 may include 2×2 array ofvibrating devices 410, 420, 430 and 440. The feedback device 130 mayprovide the user more detailed information about a type of a facialexpression and a level of a facial expression. The processor 110 maysignal feedback using the array to provide feedback depending on thetype of a facial expression.

FIGS. 4B-4G schematically depict the feedback device 130 providingdifferent types of feedback in accordance with one or more embodimentsshown and described herein. The processor 110 may signal differentfeedback based on whether the emotion of the face is a positive emotionor a negative emotion. If the emotion of the face is a positive emotion,one or more vibrating devices on the left side of the feedback device130 may vibrate. For example, when the determined emotion of a capturedimage is happiness, the processor 110 may determine the emotion aspositive, and send to the feedback device 130 a signal for activatingthe vibrating device 410 as shown in FIG. 4B. Such feedback may informthe user that the person proximate to the user is happy. When thedetermined emotion of a captured image is calmness, the processor 110may determine the emotion as positive, and send to the feedback device130 a signal for activating the vibrating device 430 as shown in FIG.4C. When the determined emotion of a captured image is happiness andlove, the processor 110 may determine the emotion as positive, and sendto the feedback device 130 a signal for activating the vibrating devices410 and 430 as shown in FIG. 4D.

If the emotion of the face is a negative emotion, one or more vibratingdevices on the right side of the feedback device 130 may vibrate. Forexample, when the determined emotion of a captured image is anger, theprocessor 110 may determine the emotion as negative, and send to thefeedback device 130 a signal for activating the vibrating device 420 asshown in FIG. 4E. Such feedback may inform the user that the personproximate to the user is angry. When the determined emotion of acaptured image is sadness, the processor 110 may determine the emotionas negative, and send to the feedback device 130 a signal for activatingthe vibrating device 440 as shown in FIG. 4F. When the determinedemotion of a captured image is anger and sadness, the processor 110 maydetermine the emotion as negative, and send to the feedback device 130 asignal for activating the vibrating devices 420 and 440 as shown in FIG.4G. The vibration intensity of each of the vibrating devices 410, 420,430 and 440 may be proportional to the level of the facial expression.

FIGS. 5A and 5B depict graphs illustrating relationship between a levelof a facial expression and a vibration intensity of the feedback device130. In FIG. 5A, the vibration intensity of the feedback device 130 islinearly proportional to the level of a facial expression. For example,if the level of smiling for a certain image is determined as 50% by theprocessor 110, the vibration intensity of the feedback device 130 isdetermined as 50% of the maximum vibration intensity. In FIG. 5B, thevibration intensity of the feedback device 130 is exponentiallyproportional to the level of a facial expression. In this embodiment,the user of the device 100 may easily recognize a change in the level ofa facial expression when the level of a facial expression is more than50% because the vibration intensity of the feedback device 130 changesmore rapidly than the vibration intensity in FIG. 5A.

It should be understood that embodiments described herein are directedto a device that provides feedback that is determined based on one ormore identifying characteristics of a subject, and varying feedbackbased on the type and/or level of a facial expression of a person. Thedevice includes an imaging device configured to capture an image of asubject, a feedback device, and a controller communicatively coupled tothe imaging device and the feedback device. The controller includes atleast one processor and at least one memory storing computer readableand executable instructions that, when executed by the processors, causethe controller to determine one or more identifying characteristics ofthe subject based on a face in the image, set one or more parameters fordetermining a facial expression based on the one or more identifyingcharacteristics of the subject, determine a facial expression in theimage based on the one or more parameters, and provide feedback with thefeedback device based on the facial expression. The device according tothe present disclosure provides more accurate information on a facialexpression of a subject by using facial recognition algorithms that arefinely tuned according to one or more identifying characteristics of thesubject being captured.

It is noted that the terms “substantially” and “proximate” may beutilized herein to represent the inherent degree of uncertainty that maybe attributed to any quantitative comparison, value, measurement, orother representation. These terms are also utilized herein to representthe degree by which a quantitative representation may vary from a statedreference without resulting in a change in the basic function of thesubject matter at issue.

While particular embodiments have been illustrated and described herein,it should be understood that various other changes and modifications maybe made without departing from the spirit and scope of the claimedsubject matter. Moreover, although various aspects of the claimedsubject matter have been described herein, such aspects need not beutilized in combination. It is therefore intended that the appendedclaims cover all such changes and modifications that are within thescope of the claimed subject matter.

What is claimed is:
 1. A device for providing feedback, the devicecomprising: an imaging device configured to capture an image of asubject; a feedback device; and a controller communicatively coupled tothe imaging device and the feedback device, the controller comprising atleast one processor and at least one memory storing computer readableand executable instructions that, when executed by the processor, causethe controller to: determine one or more identifying characteristics ofthe subject; set one or more parameters for determining a facialexpression of the subject in the image based on the one or moreidentifying characteristics of the subject; determine the facialexpression of the subject in the image based on the one or moreparameters; and provide the feedback with the feedback device based onthe determined facial expression.
 2. The device of claim 1, wherein thecomputer readable and executable instructions, when executed by theprocessor, cause the controller to determine the one or more identifyingcharacteristics of the subject based on the image from the imagingdevice.
 3. The device of claim 1, wherein the computer readable andexecutable instructions, when executed by the processor, cause thecontroller to provide the feedback with the feedback device furtherbased on the one or more parameters.
 4. The device of claim 1, furthercomprising a microphone configured to output an electrical signalindicative of a sound of the subject, wherein the computer readable andexecutable instructions, when executed by the processor, cause thecontroller to determine the one or more identifying characteristics ofthe subject further based on the electrical signal output by themicrophone.
 5. The device of claim 1, wherein the one or moreidentifying characteristics of the subject includes at least one of anidentity, a gender, an age, an ethnicity, and cultural background. 6.The device of claim 1, wherein the computer readable and executableinstructions, when executed by the processor, cause the controller toset the one or more parameters further based on a time of a day.
 7. Thedevice of claim 1, wherein the computer readable and executableinstructions, when executed by the processor, cause the controller to:determine an emotion of the subject based on the facial expression;classify the emotion as a positive emotion or a negative emotion; sendto the feedback device an instruction for vibrating a first side of thedevice with the feedback device in response to the emotion beingclassified as the positive emotion; and send to the feedback device aninstruction for vibrating a second side of the device with the feedbackdevice in response to the emotion being classified as the negativeemotion.
 8. The device of claim 1, wherein the feedback comprises atleast one of vibrations air puffs, and pressure.
 9. The device of claim1, wherein the feedback comprises a vibration pattern corresponding toMorse code.
 10. The device of claim 1, wherein the device for providingthe feedback is a smart necklace device.
 11. A device for providing afeedback, the device comprising: an imaging device configured to capturean image of a face of a subject; a feedback device; and a controllercommunicatively coupled to the imaging device and the feedback device,the controller comprising at least one processor and at least one memorystoring computer readable and executable instructions that, whenexecuted by the processor, cause the controller to: process the image ofthe face to determine one or more facial expression parameters;determine an emotion of the face based on the one or more facialexpression parameters; classify the emotion as a positive emotion or anegative emotion; send to the feedback device an instruction forvibrating a first side of the device with the feedback device inresponse to the emotion being classified as the positive emotion; andsend to the feedback device an instruction for vibrating a second sideof the device with the feedback device in response to the emotion beingclassified as the negative emotion.
 12. The device of claim 11, furthercomprising a microphone configured to output an electrical signalindicative of a sound of the subject, wherein the computer readable andexecutable instructions, when executed by the processor, cause thecontroller to determine the emotion of the face further based on theelectrical signal from the microphone.
 13. The device of claim 11,wherein the feedback device includes an array of vibration devices, andone or more vibration devices located on a first side of the feedbackdevice vibrates in response to the instruction for vibrating the firstside of the device.
 14. The device of claim 11, wherein the feedbackdevice includes an array of vibration devices, and one or more vibrationdevices located on a second side of the feedback device vibrates inresponse to the instruction for vibrating the second side of the device.15. The device of claim 11, wherein the device for providing thefeedback is a smart necklace device.
 16. A method for providing afeedback, the method comprising: capturing, by an imaging device, animage of a subject; determining, by a processor, one or more identifyingcharacteristics of the subject; setting, by the processor, one or moreparameters for determining a facial expression of the subject based onthe one or more identifying characteristics of the subject; determining,by the processor, the facial expression of the subject in the imagebased on the one or more parameters; and providing, by the processor,the feedback with a feedback device based on the determined facialexpression.
 17. The method of claim 16, wherein providing the feedbackwith the feedback device comprises providing the feedback further basedon the one or more parameters.
 18. The method of claim 16, furthercomprising: recording, by a microphone, a speech of the subject; anddetermining the one or more identifying characteristics of the subjectfurther based on the speech.
 19. The method of claim 16, wherein the oneor more identifying characteristics of the subject includes at least oneof an identity, a gender, an age, an ethnicity, and cultural background.20. The method of claim 16, further comprising: determining, by theprocessor, an emotion of the subject based on the facial expression;classifying, by the processor, the emotion as a positive emotion or anegative emotion; vibrating a first side of a device with the feedbackdevice in response to the emotion being classified as the positiveemotion; and vibrating a second side of the device with the feedbackdevice in response to the emotion being classified as the negativeemotion.